Papers with mixed-precision training
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (N19-4)
Copied to clipboard
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli
| Challenge: | OpenNMT is a community-built toolkit written in multiple languages with an emphasis on extensibility. |
| Approach: | They propose to use PyTorch to train custom sequence models for translation, summarization, language modeling, and other tasks. |
| Outcome: | The proposed toolkit is fast, extensible, and useful for both research and production. |
Towards efficient self-supervised representation learning in speech processing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models require several GPUs for days to pretrain, generating environmental concerns because of their high energy consumption. |
| Approach: | They propose an efficient self-supervised model that uses a single GPU during 24 to 48 hours of pretraining to address high computational costs. |
| Outcome: | The proposed model represents two orders of magnitude better than existing models. |